I/O Efficient Search of Large Social Networks

We introduce an I/O efficient algorithm and data structure to support fast decentralized search in large graphs modeling social networks. We structure network data in a homophily-based social hierarchy using an append-only, block-aligned skip list with an embedded tree microindex, which reduces I/O and cache line faults. We further minimize I/O when building the skip list by combining an extended memory sorting algorithm with buffered insertion. The system supports ad hoc queries efficiently: the combined cost of skip list construction and search improves performance in large social network graphs (11 million vertices and 150 million edges) by a factor of four over the fastest known external memory search algorithm and by several orders of magnitude over a traditional inmemory search algorithm. Subsequent searches that use the same skip list as an index are an order of magnitude faster still. Finally, these search structures work efficiently over a much wider range of homophily values than theory predicts.